Repository navigation
WIP WIP vulkan schema generator and struct decode/encode action implementation - #3250
Open
jzulauf-lunarg wants to merge 9 commits into
Open
jzulauf-lunarg wants to merge 9 commits into
jzulauf-lunarg wants to merge 9 commits into
Conversation
jzulauf-lunarg
requested review from
antonio-lunarg,
bradgrantham-lunarg and
fabian-lunarg
as code owners
September 6, 2026 02:19
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
jzulauf-lunarg
force-pushed
the
jzulauf-vulkan-schema-generator
branch
from
September 15, 2026 02:27
a49eebf to
d4e771f
Compare
jzulauf-lunarg
force-pushed
the
jzulauf-vulkan-schema-generator
branch
from
September 29, 2026 20:47
c766951 to
3315c60
Compare
jzulauf-lunarg
force-pushed
the
jzulauf-vulkan-schema-generator
branch
from
September 29, 2026 22:08
3315c60 to
e4e4a38
Compare
Contributor
Author
|
All Generated Decode and Encode bodies removed. Schema traits, means we're still at an increase in KLOC. Next step is removing explicit |
jzulauf-lunarg
force-pushed
the
jzulauf-vulkan-schema-generator
branch
from
October 5, 2026 17:48
0a28bde to
7e4c49d
Compare
…ery structure decoder Pilot of the Generic Field Schema and Action Model (Design-documents, consumer-graph/decode_generic_visitor_action_model.md, v7.0) for one operation family: decoding a Vulkan structure from a capture stream. What is added framework/format/format.h gains a vocabulary of logical kinds, one tag per encode-type typedef, each carrying the wire type it is recorded as. A kind is not a type: VkDeviceSize and uint64_t are the same bytes and the kind is what tells them apart. framework/schema/ is a new header-only, API-agnostic layer: field shapes, the MemberPointer trait and its Addressable/HasMember concepts, GetRef/Get/Set, the kind-and-shape concepts an Action selects on, Schema<ApiElement>::Fields, WalkFields, the command return partition and InvokeFromFields. framework/util/type_list.h supplies the TypeList facility it composes. A new generator, vulkan_schema_generator.py, emits six files from vk.xml: API type descriptors, command tags, Field descriptors and Schema specializations; the decoded-representation traits in both directions (ApiElementTraits and its inverse ApiElementFor); three member-trait partitions for native structures, decoded wrappers and decoded command args; and a compile-time checks file. Curated knowledge the registry lacks is small and named: four refined kinds (size_t, VkDeviceSize, VkDeviceAddress, VkSampleMask) and three shared descriptors (GenericHandle with its selector field, ExternalObject, OpaqueBytes). framework/decode/vulkan_decode_action.h is the one hand-written Action: twelve constrained Apply overloads covering every field shape the registry produces. Every write to the decoded value goes through Set; GetRef appears only where a decoder needs a member's address. vulkan_decode_struct_impl.h defines DecodeStruct as one constrained function template around WalkFields. ValueDecoder and PointerDecoder gain a Decode<Kind> template so the Action names kinds, not widths. What is replaced All 1,343 non-union, non-blacklisted Vulkan structures decode through the field walk. generated_vulkan_struct_decoders.cpp drops 26,000 lines of procedural bodies for one explicit instantiation per structure; one hand-written body (VkBaseOutStructure) remains and is the single entry of the NonSchemaDrivenStructs exclusion list. The forward header declares one constrained template in place of one prototype per structure. Callers are unchanged: every call still names a Decoded_T*, and a hand-written non-template overload still wins. Only that translation unit compiles the walk; the member-trait partitions reach no other target. Verification MSVC Debug and GCC 13.3 Debug build clean; 281 assertions in 19 test cases pass on both. test_vulkan_schema.cpp round-trips one structure per Apply overload against hand-encoded bytes. The checks file holds 7,309 static_asserts: every structure has a schema and no return field, every command has one return field, both trait directions round-trip, every descriptor's element type equals its wrapper's struct_type, every scalar element is as wide as its kind's wire type, and every refined kind records the same bytes as the primitive the encoder resolves it to. Regeneration reproduces the tree byte for byte. Before the procedural bodies were removed, the two decoders were compared field for field on the same buffers. Not in this change No semantic metadata is emitted. Commands have schemas, tags, traits and member traits but no decode Action; two pointer-shaped identifier overloads will be needed when they do. No capture_wrapper_type, no formal API-signature storage, no dispatch or lifecycle traits. Unions keep their hand-written decoders and are reached by legacy descent. No CI check yet enforces the private-include boundary. The capture-file smoke run and encoded-byte comparison the design's proof gates call for have not been recorded. Known latent generator limits, none reachable from the current registry: a two-dimensional static array with a Count sibling would emit an invalid extent; a pointer-returning command would emit an invalid descriptor name; base-header hierarchies (parentstruct) are excluded by is_schema_driven rather than modelled. The 26-commit history this collapses is preserved at tag jzulauf-vulkan-schema-generator-presquash. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tructure encoder, and no struct body is generated on either side
The Vulkan structure encoders join the decoders on the generated field
schema. EncodeStructAction (framework/encode/vulkan_encode_action.h) walks
a structure's Field descriptors and encodes each by shape and kind through
an adapter over ParameterEncoder; generated_vulkan_struct_encoders.cpp is
1,347 explicit instantiations and no body. The decoders reach the same
state: the one remaining generated decode body, the base-out parameter of
the ARM data-graph query, now decodes through the pNext sType sieve into
TypedStructDecoder, and its replay allocation is five lines over the
existing pNext output initializer. EncodeStruct<Struct> and
DecodeStruct<Wrapper> are plain function templates; there is no list of
structures the schema does not drive, since there are none, and a
hand-written non-template overload is preferred by overload resolution
where one exists. The generated encoders header includes the template
header, so a caller may include either, as on dev.
Schema
- one generated family split into types, fields and schema, namespaced
API first, genre second (schema::vulkan::{api_types, fields, commands})
- shapes: Value, Pointer, Array (pointer to a counted run), PointerArray
(counted run of pointers each to one element), StaticArray,
ExtensionChain, VoidReturn; concepts <Shape>ShapeField, Any<..>ShapeField,
<Kind><Shape>Field; kind families are format's to define and the field
layer lifts them, one predicate per kind
- static array extents recorded on the Field from the registry len; the
count sibling of a counted static array reaches the descriptor through
the registry's len attribute (dev PR 3265)
- counts as StoreValue operands: FieldValue<Sibling>, FieldValue<Sibling,
Member> through a pointer sibling, Constant<V>, and prefix Sum, Product,
Quotient for the registry's altlen expressions; Get returns
decltype(auto), Getter is gone, and so are 450 forward declarations
- a decoded wrapper names its own API element
Encode
- every idiom the census found has an adapter entry: scalar values and
addresses, wrapped and generic handles, text (pointer, fixed extent
bounded by the extent, and runs), fixed-extent arrays of one or two
dimensions counted or whole, pointers to one scalar, counted runs, the
extension chain from the Field's registry fact, opaque bytes, and the
kind-tag entry points that carry the named entry points' omit flags
- a counted static array is clamped by ParameterEncoder::
ClampStaticArrayLength, one implementation shared with dev's generated
callers, a template on the count's type: negative to 0, above capacity
to capacity, both reported
- the CaptureWrapperFor rows (52, one per handle descriptor) and the
DescriptorFor rows (1,347) are generated schema-family targets;
encode/vulkan_encode_descriptor_for.h holds the primary template and
the row macro
- the scalar-and-address value entry takes the member's declared type,
asserting it is the element type, an address, or a same-width integral
typedef (const void*, const CAMetalLayer*, DWORD members)
Proof
- a transitional oracle suite snapshotted all 1,343 procedural bodies and
compared bytes against the inverted library, 1,343 of 1,343 on MSVC, GCC
and clang with libc++; nothing became a canary and the suite is retired;
its schema-driven filler stays as framework/test/schema_fill.h
- capture gate against a build of the merge base: 19 traces on an NV host
and three on AMD, layer, converter and replay grabs identical block for
block, state snapshots included
- framework test: 506 assertions in 44 cases on MSVC, GCC and clang,
dev's static-array cases included; Android builds
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… VisitAt Enumerants<Enum>::entries carries each enum's values and names in registry order, generated into generated_vulkan_schema_enumerants.h; util::NameOf looks a value up through a KeyIndex. KeyIndex is a sorted constexpr index over an array, and VisitAt calls a visitor's operator()<Element> for the element at a position in a TypeList, through a table of invokers. The enum-to-string and enum-to-JSON bodies are dev's, unchanged: they stay until their generation is replaced. The enumerant test checks that ToString and to_json name every enumerant as Enumerants does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the dispatch pilot The generated 3,522-line switch and its generator go. In their place: structure_type on every typed structure descriptor (1,144, from the registry's values attribute), a descriptor for the 20 typed structures the schema excludes, and a generated catalog, catalog::structures (1,367) and catalog::extensible_structures (1,144), both sizes asserted by the checks. schema::StructureTypeIndex<List> is a static Find/End over a KeyIndex keyed by structure_type; util::IndexPosition<Index> is what Find returns and Visit takes, deducing the list from the index. The hand-written vulkan_pnext_struct_encoder.cpp keeps the loader skip and DF-2's guarded hop, and dispatches a recognized node through one Visit to EncodeStructPtr. Every structure_type is checked against GetSType, which stays this turn. WIP: three fold expressions over the catalog exceed clang's nesting limit (two static_asserts commented out, one pack constraint loosened to typename...); the flat forms and a catalog-sized ForEachType are the next turn's first item (findings section 6). The pNext test samples twelve structures until then. The framework test gets /bigobj as encode and decode already have. Gates: MSVC, GCC, clang with libc++, Android; framework 4,394/46 (pnext 30), util 7,386/38 (Linux 7,321/32). Encoder TU: 11-15 s per toolchain. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
End-of-week snapshot, MSVC only. The three fold checks disabled in 5edc3fb return in flat form: a bool array and a constexpr count, no fold. TypeListAt is Mp11's indexer (one base per element, picked by overload); the filter builds a positions array, turns it into an index_sequence once (PositionSequence), and expands over it, so no subscript is evaluated inside a template argument list. TypeListKeep, TypeListDrop and TypeListCountIf take the predicate as a value, a captureless lambda with a template call operator, the shape TypeListForEach's visitors already have; TypeListDrop is TypeListKeep over Not. IsReturnField becomes kIsReturnField; ReturnMatches is ReturnFields; TypeAt and ForEachType are TypeListAt and TypeListForEach, so every list algorithm is spelled TypeList<Op>. StructureTypeIndex loses its concept constraint so the diagnostic for a descriptor without structure_type is reachable: StructureTypesOf asserts the count, and only on failure instantiates the filtered list to name the offenders. The pNext test walks the whole catalog again; the 300-wide list in test_key_index.cpp exercises every TypeList algorithm past clang's fold limit. Not yet run: GCC, clang with libc++, Release, Android; the failure-message probe. MSVC Debug: util 7,688/39, framework 6,658/46 (pnext 2,294). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r goes util::GetSType<T>() is one template that returns the structure_type of T's API type descriptor, found through schema::binding::DescriptorFor. Its name, namespace and VkStructureType return are unchanged; two caller headers change one include line each. DescriptorFor moves from encode to schema::binding, the category of traits that join a schema entity to a C++ type existing without the schema. The rows are generated into generated_vulkan_schema_binding_descriptor_for.h, one per structure descriptor (1,367), where encode's rows covered only the described structures (1,347). Encode's static_assert names the binding. Removed: generated_vulkan_stype_util.h (1,201 lines), its generator, its registrations, and the 1,144 agreement asserts in the schema checks, which would now compare a value with itself. The shared Khronos struct-type base generator stays for OpenXR's type util. Net -2,339 lines: generated -2,324, generator Python -68, hand-written +53. Gates: MSVC Debug util 7,688/39, framework 6,658/46 (pnext 2,294); GCC Debug and clang with libc++ Debug util 7,623/33, framework 6,658/46 (pnext 2,294); Android layer and replay build. Release not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d bitvalues Follows the checkpoint a30d54d and completes the pass; the two squash later. Generated to-string bodies are gone. generated_vulkan_enum_to_string.cpp is 490 one-line explicit specializations of util/to_string.h's primaries, each a lookup over the type's enumerant table; the header keeps its 490 declarations, so callers are unchanged and dev's generic formatters (ArrayToString, BitmaskToString) still find them, which overloads declared in another header would not have. The 64-bit flag-bits types, typedefs of VkFlags64, are keyed on the API type descriptor through two templates declared in the generated header, defined in util/vulkan_enum_to_string_impl.h (included only by the .cpp) and instantiated there 32 times; dev's 32 named functions go and the util test respells. The to-string generators join the schema family. Schema: every descriptor carries name, its element type's spelling. An enumerated type's descriptor names its table (enumerants); the tables hold entries only, each an Enumerant<Descriptor>. A bits enum's descriptor carries bitmask, the type its values compose into, from the <enums> group's bitmask qualifier and bitwidth; a mask typedef's descriptor carries bitvalues, its bits descriptor, from the typedef's requires/bitvalues. One direction each, no boolean. is_enumerated replaces has_enumerants: every enum type gets a table, empty or not. catalog is its own generated header. Checks: bitmask iff the name says FlagBits; each typedef's element_type is its bits' bitmask; nothing binds uint32_t or uint64_t. Enum to JSON is dev's, by decision, until the JSON consumer is replaced. Gates, fresh binaries: MSVC Debug util 7,691/40, framework 6,658/46; GCC Debug, clang with libc++ Debug, GCC Release -Werror util 7,626/34, framework 6,658/46; Android layer and replay build. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tates its derived lists as relations
graphics/vulkan_struct_deep_copy.cpp (John) replaces the generated
vulkan_struct_deep_copy_stype switch, 1,136 cases: StructureTypeIndex over
catalog::deep_copyable_structures and one Visit into the per-structure
vulkan_struct_deep_copy. The generated switch, its generator and its
registrations are gone.
The catalog now emits only the master list and the curated data; every other
entry is a relation evaluated in C++: extensible_structures is
TypeListKeep(structures{}, kHasStructureType), with kHasStructureType beside
the concept in schema.h; deep_copy_exclusions is the structures-deep-copy key
of blacklists.json, which the deep-copy body generator also reads in place of
its literal; deep_copyable_structures is TypeListExclude of the two. The
checks file asserts each derived size against the generator's count: 9
exclusions, 1,136 deep-copyable. TypeListExclude joins type_list.h with a
300-wide test line.
MSVC Debug: util 7,691/40, framework 6,658/46. The sieve is linked and not yet
exercised by a test; its callers are encode's vulkan_track_struct.h and the
pNext hops inside the generated deep-copy bodies.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
jzulauf-lunarg
force-pushed
the
jzulauf-vulkan-schema-generator
branch
from
October 5, 2026 22:26
221397d to
1078582
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All struct decode through a central action functor
All struct encode through a central action functor